More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

摘要

Two common concerns raised in analyses of randomized experiments are (i)appropriately handling issues of non-compliance, and (ii) appropriatelyadjusting for multiple tests (e.g., on multiple outcomes or subgroups).Although simple intention-to-treat (ITT) and Bonferroni methods are valid interms of type I error, they can each lead to a substantial loss of power; whenemploying both simultaneously, the total loss may be severe. Alternatives existto address each concern. Here we propose an analysis method for experimentsinvolving both features that merges posterior predictive $p$-values forcomplier causal effects with randomization-based multiple comparisonsadjustments; the results are valid familywise tests that are doublyadvantageous: more powerful than both those based on standard ITT statisticsand those using traditional multiple comparison adjustments. The operatingcharacteristics and advantages of our method are demonstrated through a seriesof simulated experiments and an analysis of the United States Job TrainingPartnership Act (JTPA) Study, where our methods lead to different conclusionsregarding the significance of estimated JTPA effects.